{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n========================================\nRelease Highlights for scikit-learn 0.22\n========================================\n\n.. currentmodule:: sklearn\n\nWe are pleased to announce the release of scikit-learn 0.22, which comes\nwith many bug fixes and new features! We detail below a few of the major\nfeatures of this release. For an exhaustive list of all the changes, please\nrefer to the `release notes <changes_0_22>`.\n\nTo install the latest version (with pip)::\n\n    pip install --upgrade scikit-learn\n\nor with conda::\n\n    conda install scikit-learn\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "New plotting API\n----------------\n\nA new plotting API is available for creating visualizations. This new API\nallows for quickly adjusting the visuals of a plot without involving any\nrecomputation. It is also possible to add different plots to the same\nfigure. The following example illustrates :class:`~metrics.plot_roc_curve`,\nbut other plots utilities are supported like\n:class:`~inspection.plot_partial_dependence`,\n:class:`~metrics.plot_precision_recall_curve`, and\n:class:`~metrics.plot_confusion_matrix`. Read more about this new API in the\n`User Guide <visualizations>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import plot_roc_curve\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.datasets import make_classification\nimport matplotlib.pyplot as plt\n\nX, y = make_classification(random_state=0)\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n\nsvc = SVC(random_state=42)\nsvc.fit(X_train, y_train)\nrfc = RandomForestClassifier(random_state=42)\nrfc.fit(X_train, y_train)\n\nsvc_disp = plot_roc_curve(svc, X_test, y_test)\nrfc_disp = plot_roc_curve(rfc, X_test, y_test, ax=svc_disp.ax_)\nrfc_disp.figure_.suptitle(\"ROC curve comparison\")\n\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Stacking Classifier and Regressor\n---------------------------------\n:class:`~ensemble.StackingClassifier` and\n:class:`~ensemble.StackingRegressor`\nallow you to have a stack of estimators with a final classifier or\na regressor.\nStacked generalization consists in stacking the output of individual\nestimators and use a classifier to compute the final prediction. Stacking\nallows to use the strength of each individual estimator by using their output\nas input of a final estimator.\nBase estimators are fitted on the full ``X`` while\nthe final estimator is trained using cross-validated predictions of the\nbase estimators using ``cross_val_predict``.\n\nRead more in the `User Guide <stacking>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import load_iris\nfrom sklearn.svm import LinearSVC\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.ensemble import StackingClassifier\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_iris(return_X_y=True)\nestimators = [\n    ('rf', RandomForestClassifier(n_estimators=10, random_state=42)),\n    ('svr', make_pipeline(StandardScaler(),\n                          LinearSVC(random_state=42)))\n]\nclf = StackingClassifier(\n    estimators=estimators, final_estimator=LogisticRegression()\n)\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, stratify=y, random_state=42\n)\nclf.fit(X_train, y_train).score(X_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Permutation-based feature importance\n------------------------------------\n\nThe :func:`inspection.permutation_importance` can be used to get an\nestimate of the importance of each feature, for any fitted estimator:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\nfrom sklearn.inspection import permutation_importance\n\nX, y = make_classification(random_state=0, n_features=5, n_informative=3)\nrf = RandomForestClassifier(random_state=0).fit(X, y)\nresult = permutation_importance(rf, X, y, n_repeats=10, random_state=0,\n                                n_jobs=-1)\n\nfig, ax = plt.subplots()\nsorted_idx = result.importances_mean.argsort()\nax.boxplot(result.importances[sorted_idx].T,\n           vert=False, labels=range(X.shape[1]))\nax.set_title(\"Permutation Importance of each feature\")\nax.set_ylabel(\"Features\")\nfig.tight_layout()\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Native support for missing values for gradient boosting\n-------------------------------------------------------\n\nThe :class:`ensemble.HistGradientBoostingClassifier`\nand :class:`ensemble.HistGradientBoostingRegressor` now have native\nsupport for missing values (NaNs). This means that there is no need for\nimputing data when training or predicting.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.experimental import enable_hist_gradient_boosting  # noqa\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nimport numpy as np\n\nX = np.array([0, 1, 2, np.nan]).reshape(-1, 1)\ny = [0, 0, 1, 1]\n\ngbdt = HistGradientBoostingClassifier(min_samples_leaf=1).fit(X, y)\nprint(gbdt.predict(X))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Precomputed sparse nearest neighbors graph\n------------------------------------------\nMost estimators based on nearest neighbors graphs now accept precomputed\nsparse graphs as input, to reuse the same graph for multiple estimator fits.\nTo use this feature in a pipeline, one can use the `memory` parameter, along\nwith one of the two new transformers,\n:class:`neighbors.KNeighborsTransformer` and\n:class:`neighbors.RadiusNeighborsTransformer`. The precomputation\ncan also be performed by custom estimators to use alternative\nimplementations, such as approximate nearest neighbors methods.\nSee more details in the `User Guide <neighbors_transformer>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from tempfile import TemporaryDirectory\nfrom sklearn.neighbors import KNeighborsTransformer\nfrom sklearn.manifold import Isomap\nfrom sklearn.pipeline import make_pipeline\n\nX, y = make_classification(random_state=0)\n\nwith TemporaryDirectory(prefix=\"sklearn_cache_\") as tmpdir:\n    estimator = make_pipeline(\n        KNeighborsTransformer(n_neighbors=10, mode='distance'),\n        Isomap(n_neighbors=10, metric='precomputed'),\n        memory=tmpdir)\n    estimator.fit(X)\n\n    # We can decrease the number of neighbors and the graph will not be\n    # recomputed.\n    estimator.set_params(isomap__n_neighbors=5)\n    estimator.fit(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "KNN Based Imputation\n------------------------------------\nWe now support imputation for completing missing values using k-Nearest\nNeighbors.\n\nEach sample's missing values are imputed using the mean value from\n``n_neighbors`` nearest neighbors found in the training set. Two samples are\nclose if the features that neither is missing are close.\nBy default, a euclidean distance metric\nthat supports missing values,\n:func:`~metrics.nan_euclidean_distances`, is used to find the nearest\nneighbors.\n\nRead more in the `User Guide <knnimpute>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nfrom sklearn.impute import KNNImputer\n\nX = [[1, 2, np.nan], [3, 4, 3], [np.nan, 6, 5], [8, 8, 7]]\nimputer = KNNImputer(n_neighbors=2)\nprint(imputer.fit_transform(X))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Tree pruning\n------------\n\nIt is now possible to prune most tree-based estimators once the trees are\nbuilt. The pruning is based on minimal cost-complexity. Read more in the\n`User Guide <minimal_cost_complexity_pruning>` for details.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "X, y = make_classification(random_state=0)\n\nrf = RandomForestClassifier(random_state=0, ccp_alpha=0).fit(X, y)\nprint(\"Average number of nodes without pruning {:.1f}\".format(\n    np.mean([e.tree_.node_count for e in rf.estimators_])))\n\nrf = RandomForestClassifier(random_state=0, ccp_alpha=0.05).fit(X, y)\nprint(\"Average number of nodes with pruning {:.1f}\".format(\n    np.mean([e.tree_.node_count for e in rf.estimators_])))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Retrieve dataframes from OpenML\n-------------------------------\n:func:`datasets.fetch_openml` can now return pandas dataframe and thus\nproperly handle datasets with heterogeneous data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import fetch_openml\n\ntitanic = fetch_openml('titanic', version=1, as_frame=True)\nprint(titanic.data.head()[['pclass', 'embarked']])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Checking scikit-learn compatibility of an estimator\n---------------------------------------------------\nDevelopers can check the compatibility of their scikit-learn compatible\nestimators using :func:`~utils.estimator_checks.check_estimator`. For\ninstance, the ``check_estimator(LinearSVC)`` passes.\n\nWe now provide a ``pytest`` specific decorator which allows ``pytest``\nto run all checks independently and report the checks that are failing.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.utils.estimator_checks import parametrize_with_checks\n\n\n@parametrize_with_checks([LogisticRegression, DecisionTreeRegressor])\ndef test_sklearn_compatible_estimator(estimator, check):\n    check(estimator)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "ROC AUC now supports multiclass classification\n----------------------------------------------\nThe :func:`roc_auc_score` function can also be used in multi-class\nclassification. Two averaging strategies are currently supported: the\none-vs-one algorithm computes the average of the pairwise ROC AUC scores, and\nthe one-vs-rest algorithm computes the average of the ROC AUC scores for each\nclass against all other classes. In both cases, the multiclass ROC AUC scores\nare computed from the probability estimates that a sample belongs to a\nparticular class according to the model. The OvO and OvR algorithms support\nweighting uniformly (``average='macro'``) and weighting by the prevalence\n(``average='weighted'``).\n\nRead more in the `User Guide <roc_metrics>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import make_classification\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import roc_auc_score\n\nX, y = make_classification(n_classes=4, n_informative=16)\nclf = SVC(decision_function_shape='ovo', probability=True).fit(X, y)\nprint(roc_auc_score(y, clf.predict_proba(X), multi_class='ovo'))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}